Dynamic processing of storage command based on internal operations of storage system

ABSTRACT

A device and method dynamically optimize processing of a storage command within a storage system. The device and method execute a rule based on predetermined criteria and internal operation parameters of the storage system. An extended application program interface within the storage system provides internal operation parameters for use in execution of the rule. Based on execution of the rule, the storage system optimizes processing of the storage command.

FIELD OF THE INVENTION

This invention generally relates to optimizing the performance of astorage system at the pool and/or solid state drive (SSD) level.

BACKGROUND OF THE INVENTION

Nonvolatile memory devices (NVMs), such as flash memory devices, arewidely used in a variety of applications such as universal serial bus(USB) drives, digital cameras, mobile phones, smart phones, tabletpersonal computers (PCs), memory cards, and solid state drives (SSDs). Aflash memory device is typically organized into a plurality of memoryblocks, pages, and memory cells, where each of the memory blockscomprises a plurality of pages, and each of the pages comprises aplurality of memory cells.

Flash Translation Layers (FTLs) embedded in flash memory devices, suchas SSDs, facilitated the widespread adoption of flash memory devices byenabling host computers to communicate with flash memory devices viaconventional read/write hard disk drive (HDD) interfaces. To achievethis benefit, FTLs translate commands to/from a host device or remoteapplication by mapping the logical addresses used by host computer orremote application to the physical addresses of the data in the flashmemory devices. The physical data in a flash memory device is accessiblevia a low level command interface to read, program (write) and erasememory locations.

Conventional HDD interfaces employed by SSDs shield flash memory deviceoperation parameters, such as power consumption, operating temperature,and erase count, from components external to the SSD, such asapplications on a host. Host applications therefore process commandswhich arrive without any reference to the internal operations of theSSD. For example, if an incoming write command is directed to aparticular storage location that is operating at a dangerous temperature(i.e., overheating), there is currently no mechanism for processing thewrite command in a manner that avoids further taxing the overheatingstorage location.

Recent types of SSD, known as Open Channel SSDs, suffer from the sameproblems. These SSDs share data management responsibilities with thehost computers by having some of the FTL functionality implemented inthe host computer, which allows the host computer more control over theflash memory device's resources and the ability to manage multiple OpenChannel SSDs using a global FTL in the host computer. The Linux 4.4kernel is one example of an operating system kernel that supports OpenChannel SSDs, which follow the NVM Express™ specification, by providingan abstraction layer called LightNVM. However, APIs for Open ChannelSSDs still shield some information regarding a flash memory deviceoperation parameters from global FTLs.

There is, therefore, an unmet demand to optimize the performance ofnon-volatile solid state storage devices (e.g. SSDs) by dynamicallyadjusting processing of incoming commands based on internal operationinformation of SSDs.

BRIEF DESCRIPTION

In one embodiment, a method for dynamically optimizing processing of acommand within a storage system including monitoring one or moreinternal operation parameters for each of a plurality of nonvolatilememory storage locations of the storage system, receiving a command toaccess a first nonvolatile memory storage location of the plurality ofnonvolatile memory storage locations, comparing the one or more internaloperation parameters of the first nonvolatile memory storage location topredetermined internal operation criteria for the first nonvolatilememory storage location, and in response to the one or more internaloperation parameters failing to satisfy the predetermined internaloperation criteria, controlling execution of the command to access thefirst storage location in a manner to mitigate an effect of the one ormore internal operation parameters failing to satisfy the predeterminedinternal operation criteria.

In one embodiment, the controlling execution step of the method includesthrottling execution of the command to access the first nonvolatilememory storage location.

In one embodiment, the command in the method is a write or read command.

In one embodiment, the throttling execution of the method includescalculating a throttling value based on the predetermined internaloperation criteria and the one or more internal operation parameters ofthe first nonvolatile memory storage location, wherein the throttlingvalue corresponds to a deferred time to execute the command.

In one embodiment, the method further includes the step of controllingexecution comprising identifying a second nonvolatile memory storagelocation of the plurality of nonvolatile memory storage locations thatis different from the first nonvolatile memory storage location andexecuting the command to access the second nonvolatile memory storagelocation.

In one embodiment, the method includes a write command.

In one embodiment, the method includes identifying the first nonvolatilememory storage location from the command, retrieving the one or moreinternal operation parameters of the first nonvolatile memory storagelocation, and comparing the one or more internal operation parameters ofthe first nonvolatile memory storage location with the predeterminedinternal operation criteria for the first nonvolatile memory storagelocation.

In one embodiment, the method includes determining an action forexecution of the command that lowers power consumption for the firstnonvolatile memory storage location.

In one embodiment, the method includes determining an action forexecution of the command that lowers an operating temperature of thefirst nonvolatile memory storage location.

In one embodiment, the method includes one or more internal operationparameters for each of the plurality of nonvolatile memory storagelocations of the storage system includes one or more of erase count,power consumption, current or average operating temperature, and amountof fragmentation.

In one embodiment, the method includes comparing one or more internaloperation parameters for each of the plurality of nonvolatile memorystorage locations with a corresponding predetermined power, temperaturethreshold, or fragmentation value for each of a plurality of nonvolatilememory storage locations; and ranking the plurality of nonvolatilememory storage locations from a most to a least optimal storage locationfor storing data based on power or temperature.

In one embodiment, the method includes selecting the most optimalstorage location of the plurality of nonvolatile memory storagelocations from ranking the plurality of nonvolatile memory storagelocations, and executing the command to access the one of the highlyranked plurality of nonvolatile memory storage locations.

In one embodiment, the method includes one or more internal operationparameters comprising at least one of a power consumption of the storagesystem, an internal temperature of the storage system, an erase count ofthe storage system, a number of free blocks within the storage system,and a fragmentation level of the storage system.

In one embodiment, the method includes one or more internal operationparameters comprising at least one of a maximum power consumption of thestorage system, a maximum internal temperature of the storage system,and a maximum erase count of the storage system.

In one embodiment, the method includes predetermined internal operationcriteria specifying at least one of a quality-of service level of thestorage system, a service level agreement parameter of the storagesystem, and a life-time parameter of the storage system.

In one embodiment, the method includes a storage system that is asolid-state drive and non-volatile memory storage locations that arenon-volatile memory devices.

In one embodiment, the method includes a storage system that is astorage appliance and the non-volatile memory storage locations aresolid-state drives (SSDs).

In one embodiment, a device for dynamically optimizing processing of acommand within a storage system includes a plurality of non-volatilememory storage locations for storing data, a memory for storing one ormore internal operation parameters for each of the non-volatile memorystorage locations, a controller for, in response to receiving a commandto access a first non-volatile memory storage location of the pluralityof non-volatile memory storage locations, comparing the one or moreinternal operation parameter of the first nonvolatile memory storagelocation to predetermined internal operation criteria for the firstnonvolatile memory storage location, and, in response to one or moreinternal operation parameters failing to satisfy the predeterminedinternal operation criteria, controlling execution of the command toaccess the first storage location in a manner to mitigate an effect ofthe one or more operation parameters failing to satisfy thepredetermined internal operation criteria.

In one embodiment, the device includes an extended API that enables thecontroller to access internal operation parameters and predeterminedinternal operation criteria stored in the memory.

In one embodiment, the device includes non-volatile memory storagelocations that are Open Channel SSDs.

BRIEF DESCRIPTION OF THE FIGURES

FIG. 1 is a block diagram of a storage system, according to oneembodiment of the invention.

FIG. 2A is a flow chart illustrating the operation of storage system ofFIG. 1, according to one embodiment of the invention.

FIG. 2B is a flow chart illustrating the sub-steps of a step in the flowchart of FIG. 2A.

FIG. 2C is a flow chart illustrating the sub-steps of a step in the flowchart of FIG. 2A, according to one embodiment of this invention.

FIG. 3 is a block diagram of an Open Channel storage system, accordingto one embodiment of the invention.

FIG. 4A is a diagram illustrating logical address to physical locationmapping in a storage system, according to one embodiment of thisinvention.

FIG. 4B is a diagram illustrating updating logical address to physicallocation mapping information, according to one embodiment of thisinvention.

FIG. 5A is a diagram illustrating logical addresses to physical locationmapping in an Open Channel storage system, according to anotherembodiment of this invention.

FIG. 5B is a diagram illustrating updating logical address to physicallocation mapping information, according to one embodiment of thisinvention.

FIG. 6 illustrates the effect of throttling commands based on theexpected lifetime of a storage system, according to one embodiment ofthe invention.

FIG. 7 illustrates the effect of throttling commands based on the powerusage (and by extension, the temperature) of a storage location,according to one embodiment of the invention.

DETAILED DESCRIPTION

An extended application programming interface (“API”) described belowprovides access to operation parameters of a nonvolatile memory (“NVM”)based storage device (e.g., solid-state drive or SSD) or pool of SSDs(e.g., a storage appliance). Components external to the extended API,such as a global FTL, use the operational parameters to optimize theperformance of the SSD. For example, a global FTL can optimizeperformance by throttling incoming commands or balancing input/outputoperations to different storage locations within the SSD or SSDs.

Controller 101 can process additional inputs, such as predeterminedcriteria, to determine an optimizing action for processing commands. Bytaking into consideration operation parameters of an SSD as well aspredetermined criteria in processing commands, a controller for an SSDor SSDs can optimize memory performance more efficiently than prior artstorage systems thereby mitigating the effects of the operationparameters failing to satisfy the predetermined criteria. As will now bediscussed, FIGS. 1-7 below illustrate various embodiments of theextended API and using the operation parameters of an SSD for optimizingthe performance of an SSD or SSDs thereby mitigating the effects of theoperation parameters failing to satisfy the predetermined criteria.

FIG. 1 is a block diagram of storage system 100 and remote application120. Storage system 100 includes a controller 101, having aflash-translation layer (“FTL”) 110 and processor (“CPU”) 102, memory103, extended application-programming interface (“API”) 105 and storagesub-system 104 comprising, a plurality of N storage location(s) 106,where N is an integer greater than 1. In one embodiment, storage system100 is a non-volatile storage device, such as a solid-state drive(“SSD”), and storage location(s) 106 are storage locations or NVMs withthe SSD. In another embodiment, storage system 100 is a storageappliance, and storage location(s) 106 are a plurality of SSDs; in thisembodiment, storage sub-system 104 represents a pool of storagelocation(s) 106 or SSDs.

Storage system 100 includes storage subsystem 104 connected tocontroller 101. Memory 103 stores information related to operation ofthe storage system 100 including, but not limited to, metadata 103 a,operation parameter(s) 103 b, predetermined criteria 103 c, and rules103 d. Metadata 103 a can include information related to mapping storagelocation(s) 106 to virtual addresses 111. Metadata 103 a enables FTL 110to translate virtual addresses 111 (known to remote application 120) tophysical addresses in storage location(s) 106. In one embodiment, remoteapplication 120 has access to multiple K virtual addresses 111, where Kis an integer greater than 1.

In the embodiment when storage system 100 is a single SSD, virtualaddresses 111 are namespaces for accessing the storage locations 106 orNVMs of the SSD. In the embodiment when storage system 100 is a storageappliance, virtual addresses 111 are logical unit numbers (LUNs) foraccessing the storage locations 106 or individual SSDs of the storageappliance. In other words, FTL 110 uses metadata 103 a to manage theplurality of storage location(s) 106 by making them accessible to remoteapplication 120 using virtual addresses 111.

Operational parameter(s) 103 b include information related to measuredor monitored operation conditions of one or more storage location(s) 106within subsystem 104. Two or more storage location(s) 106 form a pool.For example, operation parameter(s) 103 b include: power consumption foreach storage location(s) 106; overall power consumption of a pool ofstorage location(s) 106 (more than one) or the entire storage subsystem104; historical and/or current temperature for each storage location(s)106; overall historical and/or current temperature of a pool of storagelocations 104 or the entire storage subsystem 104; a number ofprogram/erase (P/E) cycles for each storage location(s) 106; and/oramount of fragmentation for each storage location(s) 106.

The controller 101 uses the extended API 105 to calculate operationparameter(s) 103 b for a pool of storage location(s) 106 by aggregatingmeasured operation parameter(s) 103 b for each storage location(s) 106that is part of the pool. Similarly, the controller 101 uses theextended API 105 to calculate operation parameter(s) 103 b for theentire subsystem 104 by aggregating measured operation parameter(s) 103b for each storage location in subsystem 104.

Predetermined criteria 103 c includes information related to quality ofservice parameters, service level agreements, and/or expected lifetimeof individual storage location(s) 106 or a pool of storage location(s)106 within storage subsystem 104. There are multiple sources forpredetermined criteria 103 c. A user can manually define parameters.Alternatively, quality of service or service level agreements can definerequirements regarding the operation of storage system 100 based onagreed upon terms in the agreements. A service level agreement is anagreement between a client and a service provider that defines a minimumlevel of service expected from storage system 100. A service levelagreement can include required quality of service parameters such asinput/output throughput requirements, average command response time, orpriority rankings of each storage location(s) 106. A user can establishother predetermined criteria 103 c such as threshold values for theoperation parameter(s) 103 b. For example, the threshold values canapply to the storage system 100, individual storage location(s) 106, ora pool of storage location(s) 106 and can include a maximum value ofpower consumption, maximum temperature, or expected lifetime (i.e., thetime period that storage location(s) 106 are expected to work).

Rules 103 d determine how command(s) 130 are to be processed based onthe parameters within command(s) 130, related operation parameter(s) 103b, and related predetermined criteria 103 c. Controller 101 executesrules 103 d based on this information to determine optimizing actionsfor command(s) 130. As one example, execution of rules 103 d can resultin a throttling action where command(s) 130 is held (i.e., throttled)for a certain period of time before execution of the command. Thisperiod of time can be preset by the manufacturer or dependent onoperation parameter(s) 103 b. For example, if an operation parameter(s)103 b for a storage location(s) 106 a indicates the temperature ofstorage location(s) 106 a exceeds a threshold value set as apredetermined criteria 103 c, command(s) 130 directed to storagelocation(s) 106 a can be held (i.e., throttled) until the temperature ofstorage location(s) 106 a drops below the threshold value. As anotherexample, if an operation parameter(s) 103 b for storage subsystem 104indicate the overall temperature of storage subsystem 104 exceeds athreshold value set as a predetermined criteria 103 c, any command(s)130 directed to any storage location(s) 106 a in storage system 104 canbe held (i.e., throttled) until the current temperature of storagesystem 104 drops below the threshold value.

Rules 103 b can balance input/output operations of storage location(s)106 by selecting a different storage location(s) 106 that has more freespace, lower power consumption, balancing erase counts across the poolof storage location(s) 106 (i.e., pool level wear leveling), optimizepower consumption across a pool of storage location(s) 106, and/orsatisfy power consumption, temperature, and/or P/E thresholds for a poolof storage location(s) 106.

Examples of rules discussed above are exemplary and other rules directedto optimizing the performance of the storage system and storagelocations are possible

Extended API 105 enables access by controller 101 and/or FTL 110 tooperation parameter(s) 103 b of storage location(s) 106 and/or subsystem104. Controller 101 uses operation parameter(s) 103 b in processingcommand(s) 130 received from remote host 120 in order to optimizeperformance of storage location(s) 106 and/or subsystem 104.

FIG. 2A is a flow chart illustrating the operation of storage system 100according to one embodiment of the invention. In step 200, FTL 110 incontroller 101 receives command(s) 130 from a remote application 120. Instep 210, FTL 110 in controller 101 parses command(s) 130 to determinethe type of command (e.g., read or write) and the affected storagelocation(s) 106 (e.g., if a write command, the storage location(s) towhich the data is to be stored). Prior to executing command(s) 130,controller 101 optimizes execution of command (s) 130 as represented bysteps 220-260. In step 220, controller 101 retrieves operationparameter(s) 103 b associated with the affected storage location(s) 106.Extended API 105 provides access to these operation parameter(s) 103 b,which are stored in memory 103.

In step 230, controller 101 retrieves any predetermined criteria 103 cassociated with the affected storage location(s) 106. Controller 101then proceeds to retrieve any rules 103 d associated with the affectedstorage location(s) 106 in step 240. In step 250, after determination ofthe affected storage location(s) 106 and retrieval of operationparameter(s) 103 b, predetermined criteria 103 c, and rule(s) 103 d,controller 101 executes the rule(s) 103 d in step 260.

FIG. 2B is a flow chart showing the sub-steps of step 240. In sub-step240 a, controller 101 retrieves parameters from command(s) 130. Insub-step 240 c, controller 101 retrieves operation parameters 240 c. Insub-step 240 d, controller 101 retrieves predetermined criteria 103 c.In sub-step 240 b, controller 101 provides this retrieved information asinputs for rule(s) 103 d. In sub-step 240 e, execution of rule(s) 103 dresults in an action(s) that optimizes processing of command(s) 130. Asone example, the action requires throttling command(s) 130 oridentifying a new storage location(s) 106 for storing data included incommand(s) 130.

FIG. 2C provides a flow chart of one possible rule for optimizingprocessing of command(s) 130, where the command(s) 130 is a writecommand, according to one embodiment of this invention. In step 250 a,controller 101 organizes storage location(s) 106 of the storagesubsystem 104 by sorting them into a list according to at least oneparameter including erase count, amount of current power consumption,current or average operating temperature, and amount of fragmentation.After sorting, controller 101 selects a storage location(s) 106 from thesorted list in step 250 b. Controller 101 determines whether themeasured values of operation parameter(s) 103 b of the selected storagelocation(s) 106 exceed predetermined criteria 103 c (e.g., thresholds,QoS requirements) of the selected storage location(s) 106 in step 250 c.If yes, controller 101 marks the selected storage location(s) 106 andcalculates a throttling value in steps 250 d and 250 e, respectively. Asone example, controller 101 marks a storage location(s) 106 if thecurrent temperature exceeds a threshold value (saved withinpredetermined criteria 103 c) for that storage location(s) 106. In oneembodiment of step 250 e, controller 101 calculates the throttling valueaccording to the difference between a current value for an operationparameter(s) 103 b (e.g., power consumption, temperature, or erasecounts) and a threshold value (saved as a predetermined criteria 103 c)for the operation parameter(s) 103 b. As one example, controller 101calculates the throttling value by a weighted function:T=k ₁×Δ,where T is the throttling value (e.g., time to wait before executing thewrite command), Δ is the difference between a current value foroperation parameter(s) 103 b and a threshold value (saved aspredetermined criteria 103 c) for operation parameter(s) 103 b, and k₁is a predetermined weighted variable. As another example of the weightedfunction, controller 101 calculates a throttling value using a summationof the deltas of a plurality of operation parameter(s) 103 b:T _(IO) =k ₁ ×Δ+k ₂×∫Δ,where k₂ is a predetermined weighted variable and ∫Δ represents asummation of deltas.

Controller 101 can apply these weighted functions to any operationparameter(s) 103 b including power consumption, temperature, and erasecounts, to determine whether to throttle command(s) 130 or balanceinput/output (I/O) operations of a storage location(s) 106 (e.g., bydynamically selecting different storage locations to service thecommand). Throttling command(s) 130 or balancing I/O operations at astorage location optimizes performance of storage subsystem 104 byreducing the power consumption and operating temperature of storagesubsystem 104. The number of commands processed by storage location(s)106 affects the power consumption of storage location(s) 106. In otherwords, reducing (or increasing) the number of command(s) 130 processedby storage location(s) 106 results in reducing (or increasing) the powerconsumption of storage location(s) 106. Similarly, power consumption ofstorage location(s) 106 directly affects the operating temperature ofstorage location(s) 106. In other words, reducing (or increasing) powerconsumption of storage location(s) 106 results in reducing (orincreasing) the operating temperature of storage location(s) 106. Thus,throttling command(s) 130 and/or balancing I/O operations optimize thepower consumption and operating temperature of storage location(s) 106.The weighted functions above are merely examples for calculatingthrottling values, and other functions are within the scope of theinvention.

In step 250 f, if the measured values of operation parameter(s) 103 b donot exceed any thresholds (saved as predetermined criteria 103 c),storage location(s) 106 is not marked and controller 101 proceeds to thenext storage location(s) 106, if any. After selecting all storagelocation(s) 106, controller 101 determines whether there are anyunmarked storage location(s) 106 in step 250 g. If no (i.e., at leastone measured value of operation parameter(s) 103 b for all storagelocation(s) 106 exceed a threshold saved as predetermined criteria 103b), controller 101 ranks the marked storage location(s) 106 according toa weighted function in step 250 i. As one example, the weightedmulti-criteria function in step 250 i is:f ₁ =w ₁ ·OP+w ₂·power+w ₃·Temperature+w ₄·endurance+w ₅ ·IO,where w₁, w₂, w₃, w₄, and w₅ represent preset weighting values, OPrepresents the number of free pages, power represents the powerconsumption, temperature represents the operating temperature of storagelocation(s) 106, endurance represents the erase count of storagelocation(s) 106, and IO represents the calculated throttling value. Inthis example, the weighted function computes a ranking for the storagelocation(s) 106 that is inversely proportional with the number of freepages, and proportional to power consumption, operating temperature,erase count, and throttling value. After ranking of the marked storagelocation(s) 106 in step 250 i, controller 101 selects the highest rankedstorage location(s) 106 and throttles the command(s) 130 based on thethrottling value in step 250 j. After throttling, controller 101 writesthe data from the command(s) 130 to the selected storage location(s) 106in step 250 k.

In this embodiment, preference is given to unmarked storage location(s)106 (i.e., locations in which no operation parameter(s) 103 b exceeds athreshold value saved as predetermined criteria 103 c) in step 250 h.Controller 101 ranks any unmarked storage location(s) 106 using aweighted multi-criteria function. As one example, the weightedmulti-criteria function is:f ₂ =w ₁ ·OP+w ₂·power+w ₃·Temperature+w ₄·enduranceIn this example, the weighted function does not include IO as a factorsince the unmarked storage locations do not have a throttling value.Controller 101 selects a highest-ranking storage location(s) 106 andwrites the data from the command(s) 130 to the selected storage location160 in step 250 k.

FIG. 3 is a block diagram of a storage system that implements storagelocations 306 as Open Channel SSDs. Open Channel SSDs are SSDs that maynot have all the FTL implemented within the device. Instead, the remotehost or application 320 includes FTL 310. In this embodiment, acontroller 301 includes a CPU 302 and an extended Open Channel API 305that enables the host FTL 310 to access operation parameters 303 bstored in memory 303. Extended Open Channel API 305 provides internaloperation parameters of storage system 300, such as power consumptionand operating temperature of storage system 300. Open Channel device 306within a storage subsystem 304 each include a standard Open Channel API305, through which the controller 301 interfaces with the Open Channeldevices.

FIGS. 2A-2C describe steps performed by systems 100 of FIG. 1 and system300 of FIG. 3 to throttle commands as one example of an optimizingaction for processing commands.

FIGS. 4-7 provide another example of an optimizing action and illustratebalancing I/O operations through selection of a different storagelocation for executing commands. While the following discussionreferences system 100 of FIG. 1, system 300 of FIG. 3 also is capable ofperforming the steps below.

FIG. 4A illustrates a logical address to physical location mapping instorage system 100 according to one embodiment of this invention. Inthis embodiment, storage system 100 is an SSD. In another embodiment,storage system 100 can be a storage appliance. Virtual addresses 400include segments called fixed size chunks 401. In one embodiment,virtual addresses 400 are LUNs. In another embodiment, virtual addresses400 are namespaces. For compatibility with NAND memory geometry, thechunk size is 4 kilobytes.

Each virtual chunk 401 includes a descriptor containing a mapping 405 toa physical storage location 410. Mapping information 405 includes anidentification (SL #) of a storage location 410 and a logical blockaddress (LBA) 402 within storage location 410. Mapping information 405can be stored as metadata and is accessible by controller 101. Mappinginformation 405 in each virtual chunk 401 points 420 to a correspondingLBA 402 within a storage location 410. In an embodiment where storagesystem 100 is a storage appliance, storage location 410 is an SSD andthe physical storage location identification is an SSD ID.

As described above, an optimizing action for processing write command(s)130 can cause selection of a SSD 410 that is different from current SSDstoring command(s)′ data. In one embodiment, command(s) are writecommands. Selecting a different SSD 410 for placing data requiresdynamically updating the mapping (represented by pointers 420) betweenvirtual addresses 400 to SSD 410. That is, controller 101 can update thephysical location of data associated with a virtual chunk 401 based onoperation parameter(s) 103 b of SSD 106.

FIG. 4B illustrates the process of updating mapping information 405between a logical and physical address after selection of a differentSSD 106 during processing of command(s) 130. Prior to receivingcommand(s) 130, virtual chunk 401 in virtual address 400 originallycontains mapping information 405 to an LBA 402 a within a SSD 410 a.Upon receiving an incoming command that requires processing (e.g.,writing) data of virtual chunk 401, controller 101 uses mappinginformation 405 to identify LBA 402 a to which the data is to bewritten. In one embodiment, after executing a rule(s) as describe above,controller 101 can select a different LBA 402 b in SSD 410 b and writesthe data to LBA 402 b. This action can be a result of controller 101determining that SSD 410 a exceeds a threshold value for powerconsumption, operating temperature, or number of erase counts.Controller 101 places data from command(s) 130 in the new locationassociated with LBA 402 b in SSD 410 b. In addition, controller 101modifies the descriptor of virtual chunk 401 such that it points 425 toLBA 402 b in SSD 410 b. A TRIM command is sent to SSD 410 a to evictdata at address 402 b.

FIG. 5A illustrates a virtualization mechanism mapping logical addressesto physical locations in an Open Channel storage system according toanother embodiment of this invention. In this embodiment, storage system100 is an Open Channel SSD. Virtual addresses 500 includes segmentscalled fixed-size virtual chunks 501. For compatibility with NANDgeometry, the chunk size is 4 kilobytes, Thus, every virtual chunk isstored in a physical NAND page.

Each chunk 501 includes a descriptor containing a mapping 505 fromvirtual chunk 501 to a physical location in Open Channel device 515.Mapping information 505 includes an identification (OC #) of OpenChannel device 515, a block number (BLOCK#) within Open Channel SSD 515,and a page number (PAGE #) within the block number. Controller 101 canaccess mapping information 505 can be stored as metadata in memory.Mapping information 505 within each virtual chunk points 520 to acorresponding page 502 within a block 510 within an Open Channel SSD515.

FIG. 5B illustrates the process of updating mapping information 505between a logical and physical address after selection of a differentstorage location during processing of an incoming command for an OpenChannel SSD. Prior to receiving an write command, virtual chunk 501 invirtual address 500 a originally contains mapping information 505 to anOpen Channel device 515 a, block 510 a within Open Channel device 515 a,and page 502 a within block 510 a. Upon receiving the command thatrequires processing (e.g., writing) data of virtual chunk 501,controller 101 uses mapping information 505 to identify the block 510 aand page 502 a within Open Channel SSD 515 to which the data is to bewritten. In one embodiment, controller 101 selects a different page 502b in block 510 b, and subsequently write data to page 502 b. This actioncan result from determining that device 515 a exceeds a threshold valuefor power consumption, operating temperature, or number of erase counts.Controller 101 can place data from the command in new page 502 b indevice 510 b. In addition, controller 101 modifies the descriptor ofvirtual chunk 501 such that it points 525 to page 502 b. Invalid datalist 530 stores old mapping information including page block 510 a andpage 502 a. At a later point, a garbage collection operation clears datalocated in page 502 a of block 510 a and a trim command informs a remotehost of the removal of data.

FIG. 6 shows the effect of throttling commands over a period of time,according to one embodiment of the invention. A storage system has alimited number of P/E cycles. Line 600 represents the actual P/E cyclesof a storage device and dotted line 610 is the threshold P/E cycles forthe storage device that is necessary to meet the expected lifetime forthe storage device. A storage system according to one embodiment of thecurrent invention executes rules to optimize write commands to controlthe P/E cycles of storage locations. When a value of the actual P/Ecycles 600 exceeds the threshold P/E value 610 for a certain storagelocation, controller 101 executes rules to provide optimizing actionsthat throttle commands directed at the storage location. In oneembodiment, the rule-based actions are triggered when the actual P/Ecycles exceeds the threshold by a certain delta difference, Δ. Acontroller uses this delta to calculate a throttling value as discussedabove.

FIG. 7 shows the effect of throttling commands based on the power usage(and by extension, the temperature) of a storage location. Line 700represents actual power consumption of a storage location and dottedline 710 represents a predetermined threshold for the power consumptionof the storage location. When a value of actual power consumption 700exceeds the power consumption threshold 710, controller 101 canimplement the rule-based actions as described above which results inoptimization of the command. Optimization of the command includes, forexample, throttling the command or selecting a different storagelocation for processing the command. The difference, Δ, between theactual and threshold power consumption, can be used to calculate athrottling value for the command, as discussed above.

Other objects, advantages and embodiments of the various aspects of thepresent invention will be apparent to those who are skilled in the fieldof the invention and are within the scope of the description and theaccompanying figures. For example, but without limitation, structural orfunctional elements might be rearranged, or method steps reordered,consistent with the present invention. Similarly, principles accordingto the present invention could be applied to other examples, which, evenif not specifically described here in detail, would nevertheless bewithin the scope of the present invention.

What is claimed is:
 1. A method for dynamically optimizing processing ofa command within a storage system, comprising: monitoring one or moreinternal operation parameters for each of a plurality of nonvolatilememory storage locations of the storage system; receiving a command toaccess a first nonvolatile memory storage location of the plurality ofnonvolatile memory storage locations; comparing the one or more internaloperation parameters of the first nonvolatile memory storage location topredetermined internal operation criteria for the first nonvolatilememory storage location; and in a manner to mitigate an effect of theone or more internal operation parameters failing to satisfy thepredetermined internal operation criteria, deferring execution of thecommand to access the first storage location until the one or moreinternal operation parameters satisfy the predetermined internaloperation criteria, or executing the command to access a secondnonvolatile memory storage location instead of the first storagelocation.
 2. The method according to claim 1, wherein the step ofdeferring execution comprises throttling execution of the command toaccess the first nonvolatile memory storage location.
 3. The methodaccording to claim 2, wherein the command is a write or read command. 4.The method of claim 2, wherein throttling execution comprises:calculating a throttling value based on the predetermined internaloperation criteria and the one or more internal operation parameters ofthe first nonvolatile memory storage location, wherein the throttlingvalue corresponds to a deferred time to execute the command.
 5. Themethod according to claim 1, wherein the method comprises, in the mannerto mitigate the effect of the one or more internal operation parametersfailing to satisfy the predetermined internal operation criteria,executing the command to access a second nonvolatile memory storagelocation; and identifying the second nonvolatile memory storage locationof the plurality of nonvolatile memory storage locations that isdifferent from the first nonvolatile memory storage location.
 6. Themethod according to claim 5, wherein the command is a write command. 7.The method according to claim 1, further comprising: identifying thefirst nonvolatile memory storage location from the command; retrievingthe one or more internal operation parameters of the first nonvolatilememory storage location; and comparing the one or more internaloperation parameters of the first nonvolatile memory storage locationwith the predetermined internal operation criteria for the firstnonvolatile memory storage location.
 8. The method according to claim 7,further comprising determining an action for deferring or executing ofthe command that lowers power consumption for the first nonvolatilememory storage location.
 9. The method according to claim 7, furthercomprising determining an action for deferring or executing of thecommand that lowers an operating temperature of the first nonvolatilememory storage location.
 10. The method according to claim 1, whereinthe one or more internal operation parameter for each of the pluralityof nonvolatile memory storage locations of the storage system includesone or more of erase count, power consumption, current or averageoperating temperature, and amount of fragmentation.
 11. The methodaccording to claim 10, further comprising: comparing the one or moreinternal operation parameters for each of the plurality of nonvolatilememory storage locations with a corresponding predetermined power,temperature threshold, or fragmentation value for each of a plurality ofnonvolatile memory storage locations; and ranking the plurality ofnonvolatile memory storage locations from a most to a least optimalstorage location for storing data based on power or temperature.
 12. Themethod according to claim 11, further comprising: selecting the mostoptimal storage location of the plurality of nonvolatile memory storagelocations from ranking the plurality of nonvolatile memory storagelocations; and executing the command to access the one of the highlyranked plurality of nonvolatile memory storage locations.
 13. The methodof claim 1, wherein the one or more internal operation parameterscomprise at least one of a power consumption of the storage system, aninternal temperature of the storage system, an erase count of thestorage system, a number of free blocks within the storage system, and afragmentation level of the storage system.
 14. The method of claim 1,wherein the one or more internal operation parameters comprise at leastone of a maximum power consumption of the storage system, a maximuminternal temperature of the storage system, and a maximum erase count ofthe storage system.
 15. The method of claim 1, wherein the predeterminedinternal operation criteria specifies at least one of a quality-ofservice level of the storage system, a service level agreement parameterof the storage system, and a life-time parameter of the storage system.16. The method of claim 1, wherein the storage system is a solid-statedrive and the non-volatile memory storage locations are non-volatilememory devices.
 17. The method of claim 1, wherein the storage system isa storage appliance and the non-volatile memory storage locations aresolid-state drives (SSDs).
 18. The method of claim 1, the method furthercomprising: determining whether, in the manner to mitigate the effect,to perform: deferring execution of the command to access the firststorage location, or executing the command to access the secondnonvolatile memory storage location instead of the first storagelocation.
 19. A device for dynamically optimizing processing of acommand within a storage system, comprising: a plurality of non-volatilememory storage locations for storing data; a memory for storing one ormore internal operation parameters for each of the non-volatile memorystorage locations; a controller configured to: in response to receivinga command to access a first non-volatile memory storage location of theplurality of non-volatile memory storage locations, comparing the one ormore internal operation parameter of the first nonvolatile memorystorage location to predetermined internal operation criteria for thefirst nonvolatile memory storage location, and, in a manner to mitigatean effect of the one or more internal operation parameters failing tosatisfy the predetermined internal operation criteria, deferringexecution of the command to access the first storage location until theone or more internal operation parameters satisfy the predeterminedinternal operation criteria, or executing the command to access a secondnonvolatile memory storage location instead of the first storagelocation.
 20. The device according to claim 19, the controller furthercomprising an extended API that enables the controller to accessinternal operation parameters and predetermined internal operationcriteria stored in the memory.
 21. The device according to claim 19,wherein each non-volatile memory storage location is an Open ChannelSSD.
 22. The device according to claim 19, wherein the controller isfurther configured to: determine whether, in the manner to mitigate theeffect, to perform: deferring execution of the command to access thefirst storage location, or executing the command to access the secondnonvolatile memory storage location instead of the first storagelocation.